Action Recognition using ST-patch Features for First Person Vision

Much research has been devoted in recent years to recognizing human action from video images. Most existing methods, however, take video of people from the outside making it difficult to understand behavioral intention. The First Person Vision approach has been proposed in response to this problem. In this approach, a device consisting of two cameras collectively called an “inside-out camera” is attached to the head of a person to obtain “scene video” that captures the person’s visual field and “eyeball video” that observes one of the person’s eyeballs. These video streams are used as a basis for understanding that person’s behavioral intention. With the aim of realizing First Person Vision, we here calculate features using a global ST-patch and a local ST-patch from scene video and attempt to distinguish six types of actions while walking using Joint Boosting. Results of a comparison experiment revealed that the proposed method improved the accuracy of distinguishing actions by 27.3% compared to an optical-flow method.